Almost all problems in signal processin and machine learning, after problem formulation, we resort to optimization method to find a solution. I focus on un-constranied optimization problem and constrained optimization problem for convex functions. Then, I will add some result on non-convex optimization. I plan to cover:

Part I (basic)
1. Convex function and gradient method and conjugate gradient method
2. Convergence rate and strongly convex function
3. Conjugate function and subgradient
4. Directional derivation, subgradient calculation and method
5. Proximal point method and proximal gradient method

Part II (basic)

6. Projected gradient method and Penalty method
7. Lagrangian method and augmented Lagrangian method
8. Block coordinate descent method and alternating direction method of multipliers

Part III: beyond convex optimizations
9. Variational inequality for non-convex optimization
10. PALM algorithm
11. Liearlization methods